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Background 


Health human resources—physicians, nurses 
and other health-care providers—are crucial to 
the delivery of health services. They represent 
the single greatest asset, as well as cost, to the 
health-care system. Acting to address concerns over 
provincial physician and nursing shortages, long 
wait times and an increasing number of patients 
without family doctors, the Ministry of Health and 
Long-Term Care and the Ministry of Training, Col¬ 
leges and Universities jointly developed a strategy 
called HealthForceOntario in the 2005/06 fiscal 
year. As part of the strategy, the Ministry of Health 
and Long-Term Care established the HealthForce- 
Ontario Marketing and Recruitment Agency 
(Agency) in 2007. The Agency’s activities focus on 
recruitment and retention of health professionals. 

The strategy’s goal is to ensure that Ontarians 
have access to the right number, mix and distribu¬ 
tion of qualified health-care providers, now and in 
the future. Responsibility for its implementation 
lies with the Health Human Resources Strategy 
Division of the Ministry of Health and Long-Term 
Care (Ministry), but its Assistant Deputy Minister 
reports to the Deputy Ministers at both ministries. 
This is meant to establish a link between the health 


care and education systems, and better manage the 
supply of health human resources. 

Total expenditures for the strategy grew 
from $448 million in the 2006/07 fiscal year 
to $738.5 million in the 2012/13 fiscal year, an 
increase of about 65%. These amounts included 
$431 million for physician and nursing initiatives in 
2006/07 and $728 million for them in 2012/13, as 
well as ministry operating expenses of $17 million 
in 2006/07 and $10.5 million in 2012/13, as shown 
in Figure 1. 


Audit Objective and Scope 


The objective of our audit was to assess whether the 
Ministry of Health and Long-Term Care, in conjunc¬ 
tion with the Agency, had adequate systems and 
procedures in place to: 

• identify and assess the appropriateness of 
the mix, supply and distribution of qualified 
health-care professionals to help meet the cur¬ 
rent and future needs of Ontarians across the 
province; 

• ensure that strategy initiatives were delivered 
in accordance with established regulatory 
requirements, applicable directives and poli¬ 
cies, and agreements; and 
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Figure 1: Health Human Resources Strategy Division 
Expenditures, 2006/07-2012/13 ($ million) 

Source of data: Ministry of Heaitii and Long-Term Care 



• measure and report regularly on the progress 
of the strategy’s objectives. 

The Ministry and Agency senior management 
reviewed and agreed to our audit objective and 
criteria. 

Our audit focused on physician and nurse human 
resources. In conducting our audit, we reviewed 
relevant legislation, administrative policies and pro¬ 
cedures, and interviewed staff at the Ministries of 
Health and Long-Term Care and Training, Colleges 
and Universities. We visited three Local Health Inte¬ 
gration Network (LHIN) offices and three hospitals 
in the North West and South West regions, and we 
contacted two hospitals in the Greater Toronto Area 
region to interview staff and obtain relevant docu¬ 
ments. We also obtained information related to vari¬ 
ous nursing initiatives from Greater Toronto Area 
hospitals. To gain an overall understanding and 
perspective of the health human resources area, we 
spoke with a number of external stakeholders such 
as the Gollege of Family Physicians, the Ontario 
Hospital Association, the Registered Nurses Associa¬ 
tion of Ontario, the Registered Practical Nurses 
Association of Ontario, the Professional Association 
of Residents of Ontario (formerly the Professional 
Association of Internes and Residents of Ontario), 
and the Ontario Medical Association. We also spoke 


to representatives of other jurisdictions—Manitoba, 
Alberta and British Golumbia—to gain an under¬ 
standing of how health human resource planning is 
done in those provinces. 


Summary 


Over the last six years, the Ministry of Health and 
Long-Term Gare (Ministry) has spent $3.5 b illi on 
through its HealthForceOntario strategy to address 
the shortages of physicians, nurses and other health 
professionals across Ontario. In 2012/13 the Min¬ 
istry directed $738.5 million toward this strategy: 
$505 million for physician initiatives, $151 million 
for nursing initiatives, $72 million for other health 
human resource initiatives and the remaining 
$10.5 million for operating expenses. 

Overall, Ontario has seen an 18% increase in 
physicians from 2005 to 2012 and a 10% increase 
in nurses from 2006 to 2012. While the initiatives 
increased enrolment, created more postgraduate 
training positions and attracted more doctors and 
nurses from other jurisdictions, Ontario has not met 
its goal of having the right number, mix and distri¬ 
bution of physicians in place across the province to 
meet the population’s future health-care needs. 

Specifically, we noted the following: 

• The province spends an average of about 
$780,000 (including $375,000 for resident 
salaries and benefits) to educate one special¬ 
ist who completes a four-year undergraduate 
degree and up to five years of postgraduate 
residency training. For a specialist who 
enters Ontario at the postgraduate level from 
outside the province, this cost is $225,000. 
However, many specialists trained in Ontario 
do not stay and practise here. Retention 
statistics show that, on average, 33% of 
Ontario-funded surgical specialist graduates 
left the province each year between 2005 
and 2011. The lack of full-time employment 
opportunities for graduating residents of 
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certain surgical specialties may lead to more 
physicians deciding to leave the province, 
despite long wait times for these services. For 
example, wait-time data for the three-month 
period from June to August 2013 showed 
waits of 326 days for forefoot surgery and 
263 days for cervical disc surgery. 

• The Agency provides temporary physician 
or “locum” coverage in eligible communities 
across the province to support access to care. 
However, vacancy-based locum programs 
meant as short-term measures continued to 
be used for long periods of time. At the time 
of our audit there were about 200 specialist 
vacancies in Northern Ontario, and of those 
hospitals using locum services, one-third that 
had been using the Emergency Department 
Coverage Demonstration Project before Janu¬ 
ary 2008 had been continuously using its 
locum services from as early as 2007, and one 
hospital had been using them since 2006. 

• Over the four fiscal years from 2008/09 to 
2011/12, $309 million was dedicated to hir¬ 
ing 9,000 new nurses. Our review showed 
that while the system was unable to hire that 
many nurses in the four years, it had increased 
the number of nurses by more than 7,300 and 
the Ministry was on track to achieve its goal 
within five years. 

• At the end of 2011, 66.7% of nurses were 
working full-time in Ontario, which was just 
slightly under the Ministry’s goal of 70% of 
nurses working on a full-time basis. However, 
the Ministry needed to improve its oversight 
and assessment of the effectiveness of its nurs¬ 
ing programs and initiatives. For example, 
funding for the Nursing Graduate Guarantee 
Program is provided for up to six months with 
the expectation that organizations will offer 
permanent full-time employment for partici¬ 
pating new graduate nurses. However, only 
about one-quarter of program participants in 
2010/11 and one-third in 2011/12 actually 
obtained permanent full-time positions. 


• Although the physician forecasting model 
built in partnership with the Ontario Medical 
Association was a positive step in determining 
physician workforce requirements, it is ham¬ 
pered by the limited reliability and availability 
of data. These limitations make planning the 
optimal number, mix and distribution of phys¬ 
icians with appropriate funding, training and 
deployment difficult. As well, a simulation 
model being developed by the Ministry to help 
plan for future nursing education positions 
and to help formulate nursing policies aimed 
at recruitment and retention determines only 
what the supply of nurses will be without 
considering how many nurses will be needed 
to meet the population’s needs. 

B OVERALL MINISTRY RESPONSE 

The Ministry of Health and Long-Term Gare 
(Ministry) and the HealthForceOntario Market¬ 
ing and Recruitment Agency acknowledge and 
thank the Auditor General for the timely audit 
and the recommendations in this report. 

In a Ganadian first, the province launched 
the HealthForceOntario strategy in May 2006. 
This was an innovative response to existing 
critical shortages in health human resources, 
and it aimed to ensure that existing gaps would 
not worsen. 

The strategy has led to a significant improve¬ 
ment in the health human resource capacity of 
Ontario. Shortages of health providers, includ¬ 
ing physicians and nurses, are no longer the 
primary barrier to access or cause of wait times. 
The strategy has mitigated the shortages and 
improved the province’s ability to plan, train 
and support its health workforce, with some key 
results since May 2006 including: 

• more than 35,000 new regulated providers, 
including an 18% increase in physician sup¬ 
ply and a 10% increase in nurse supply; 

• expanded first-year undergraduate enrol¬ 
ment in medical schools (up by 22%) and 
first-year postgraduate trainees (up by 60%); 
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• 15,644 more nurses working full-time, a 
23% improvement; 

• 25 nurse practitioner-led clinics providing 
care to over 36,000 patients; 

• more than 15,100 employment opportun¬ 
ities for new Ontario nursing graduates; 

• new health-care provider roles including 
physician assistants, clinical specialist radia¬ 
tion therapists and five new nursing roles; 

• creation of evidence capacity to inform 
planning; 

• legislative and regulatory changes increas¬ 
ing the quality and safety of patient care, 
expanding scopes of practice and regulating 
new health professions; and 

• establishment of the HealthForceOntario 
Marketing and Recruitment Agency. 

Ontario is now able to focus health human 

resource activities on health-system transforma¬ 
tion rather than responding to critical shortages 
of providers. The Ministry’s work continues 
to evolve to address today’s challenges. The 
Ministry is renewing the HealthForceOntario 
strategy so that it: 

• builds on the successes of previous 
accomplishments; 

• aligns with the goals of Ontario’s Action Plan 
for Health Care; and 

• advances evidence-informed planning and 
decision-making. 

The recommendations in this audit will 
inform the strategy renewal. 


Detailed Audit Observations 


PHYSICIANS 

Over the last six years, the Ministry of Health and 
Long-Term Care (Ministry) has spent $3.5 billion 
through its HealthForceOntario strategy to address 
the shortages of physicians, nurses and other health 


professionals across Ontario. While the province 
was able to increase the number of physicians, some 
Ontario communities face shortages of health-care 
providers, especially physicians. Primary-care 
physicians, also known as family physicians, are 
not always available in small, rural or remote com¬ 
munities. In Northern Ontario, general specialists 
(for example, in the areas of general surgery, inter¬ 
nal medicine and psychiatry) also remain in high 
demand despite a significantly improved provincial 
physician supply. 

Although the significant amount of funds that 
the Ministry has expended over the last six years 
has increased the supply of physicians in the prov¬ 
ince, shortages remain in certain specialties and 
geographical areas even as physicians in those spe¬ 
cialties are unable to obtain full-time employment. 

Increased Supply of Physicians in Ontario 

Medical education is funded jointly by the Min¬ 
istry of Training, Colleges and Universities and 
the Ministry of Health and Long-Term Care. The 
Ministry of Training, Colleges and Universities 
funds universities for undergraduate positions, 
while the Ministry of Health and Long-Term Care 
funds most aspects of postgraduate training. The 
majority of the $485 million the Ministry of Health 
and Long-Term Care spent on physician initiatives 
in the 2011/12 fiscal year was in two areas: total 
payments of $315 million to medical schools and 
hospitals for the salaries and benefits of residents 
who provide clinical services across Ontario; and 
$107 million paid to medical schools to support 
academic activities such as teaching, educational 
infrastructure and related administrative costs for 
clinical education of medical learners. 

From 2005 to 2012, the Ministry of Health and 
Long-Term Care worked with the Ministry of Train¬ 
ing, Colleges and Universities to increase enrol¬ 
ment in physician training programs. First-year 
undergraduate enrolment in medical schools went 
up by 22% and first-year postgraduate trainees by 
60%. In family medicine, the number of first-year 
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postgraduate trainees went up by 67% and spe¬ 
cialists by 56%. In addition, the number of inter¬ 
national medical graduates who entered residency 
training went up by 48%. As seen in Figure 2, 
between 2005 and 2012, the number of physicians 
increased by 18%, or about 4,100. At the same 
time, the number of family doctors per 100,000 
people went from 84.9 to 91, and specialists from 
92.9 to 104.3. The total number of doctors per 
100,000 people went from 177.8 to 195.3. Accord¬ 
ing to the Canadian Institute for Health Informa¬ 
tion, the number of specialists per 100,000 people 
in Ontario in 2011 was in line with the Canadian 
average, while the number of family physicians per 
100,000 people in Ontario was about 10% below 
the average. 

Sub-optimal Distribution of Physicians in 
Ontario 

Despite the overall increase in primary health-care 
providers and specialists, access to health care is 
still a problem for some Ontarians. According to the 
Ministry, based on data collected between October 
2012 and March 2013, 6% of Ontarians lacked a 
family physician. Although more recent regional 
data is not available, in 2010 the percentage in the 


north was twice as high. Access has been a long¬ 
standing issue in many rural, remote and northern 
communities in Ontario with chronic physician 
shortages. Geographic isolation, long travel dis¬ 
tances, low population densities and inclement 
weather conditions are just some of the challenges 
to providing health care in these areas. 

A 2011 Canadian Institute for Health Informa¬ 
tion report showed that 95% of physicians in 
Ontario practised in urban areas while the remain¬ 
ing 5% practised in rural areas. This number falls 
short of urban-rural population distribution in 
Ontario: according to Statistics Canada’s 2011 
census, 86% of the population lived in urban areas 
while 14% lived in rural areas. To help assess 
the accessibility of health care in rural areas, the 
Ministry uses the Rurality Index of Ontario (RIO), 
developed by the Ontario Medical Association. The 
RIO incorporates data on population and physicians 
practising in rural and northern areas, including 
large urban centres in the north. The RIO indicates 
that in 2011, 8.1% of physicians in Ontario prac¬ 
tised in these areas, which contained 11.6% of the 
province’s population. 

Although the Ministry acknowledged that 
physician distribution across Ontario was still not 
optimal, it cited factors that could account for the 


Figure 2: Increase in the Number of Physicians and Physician Trainees in Ontario, 2005-2012 

Source of data: Ministry of Health and Long-Term Care 



2005 

2012 

Increase (%) 

First-year undergraduate enrolment 

797 

972* 

22 

Medical school graduates 

663 

875 

32 

First-year postgraduate trainees 

757 

1,213 

60 

Family medicine - first-year postgraduate trainees 

305 

508 

67 

Specialty - first-year postgraduate trainees 

452 

705 

56 

International medical graduates 

171 

253 

48 

Family physicians 

10,641 

12,296 

16 

Specialists 

11,636 

14,086 

21 

Total physicians 

22,277 

26,382 

18 

Family medicine physicians per 100,000 population 

84.9 

91.0 

7 

Specialists per 100,000 population 

92.9 

104.3 

12 

Total physicians per 100,000 population 

177.8 

195.3 

10 


* Latest data available for undergraduate enrolment is from 2011. 
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way physicians are distributed. For example, some 
highly specialized health-care services are delivered 
in tertiary care units, which means patients in some 
communities must travel to large urban centres to 
receive specialized care. 

Medical Specialties Facing Employment 
Problems 

On average, the province invests about $780,000 
(including $375,000 for resident salaries and 
benefits) to educate one specialist for a four-year 
undergraduate degree and up to five years of post¬ 
graduate residency training. For a specialist who 
enters Ontario from outside the province at the 
postgraduate training level, this cost is $225,000. 

In 2011, the province spent a total of $438 million 
on specialist education—a 63% increase since 
2005, when the amount spent was $269 million. 
However, many specialists trained in Ontario 
do not stay and practise here. Figure 3 shows 
that, on average, about 33% of surgical specialist 
graduates (including neurosurgeons and cardiac, 
orthopaedic, paediatric and general surgeons) who 
were funded by the Ministry left Ontario each year 
between 2005 and 2011. 

Not every graduating specialist who leaves 
Ontario does so because of employment difficul¬ 
ties. The size and breadth of Ontario postgraduate 


training programs may attract trainees from juris¬ 
dictions where these programs are not offered, and 
some may complete residency training in Ontario 
and return to their home province afterward. 
However, others may leave Ontario because they 
have difficulty finding stable employment after 
graduation. 

The Royal College of Physicians and Surgeons of 
Canada (College) and the National Specialties Soci¬ 
eties conducted a mini-study from July to Novem¬ 
ber 2010 that found physician unemployment and 
underemployment were common in the following 
areas: cardiac surgery, nephrology, neurosurgery, 
plastic surgery, public health and preventative 
medicine, otolaryngology (ear, nose and throat 
specialists) and radiation oncology. In light of these 
results, the College expanded its research in April 
2011 to conduct a multi-phase national study of 
medical specialist employment in Canada. The final 
report was released in October 2013. The report 
indicated that the specialties affected included 
orthopaedic surgery, urology, gastroenterology, 
hematology, critical care, general surgery, ophthal¬ 
mology, neurosurgery, nuclear medicine, otolaryn¬ 
gology and radiation oncology. In our discussions 
with medical associations, we repeatedly heard that 
graduating specialists face employment difficulties 
in various surgical specialties, including many of 
the above. 


Figure 3: Surgical Specialists Leaving Ontario, 2005-2011 

Source of data: Ministry of Health and Long-Term Care retention data extracted from Canadian Post-M.D. Education Registry (CAPER) 
Annual Census of Post-M.D. Trainees 


Average for 



2003 

2004 

2005 

2006 

2007 

2008 

2009 

2003-09 

Surgical specialists graduating in Ontario 

120 

122 

120 

125 

111 

114 

147 

123 



Average for 


2005 

2006 

2007 

2008 

2009 

2010 

2011 

2005-11 

Surgical specialists practising in Ontario two years 
after graduation 

79 

79 

79 

87 

73 

70 

108 

82 

Surgical specialist graduates leaving Ontario 

(41) 

(43) 

(41) 

(38) 

(38) 

(44) 

(39) 

(41) 

% of surgical specialist graduates leaving Ontario 

34 

35 

34 

30 

34 

39 

27 

33 

Surgical specialists coming into Ontario to practise 

22 

8 

12 

19 

18 

14 

9 

15 

Net number of specialists leaving Ontario 

19 

35 

29 

19 

20 

30 

30 

26 
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The College’s 2011-12 study found that about 
20% of new specialists and subspecialists in Ontario 
(compared to 16% in Canada) could not find a job 
after completing their residency training periods of 
two to five years following medical school. It also 
noted that employability was impacted by personal 
factors and preferences such as jobs not being based 
in new graduates’ preferred locations; hospital 
budgetary restrictions; and delayed retirements. 

Such factors could result in physicians choosing to 
prolong their studies or make do with contract and 
temporary work, losing skills, leaving Ontario or 
having to work in non-surgical practice. At the same 
time, there are specialist vacancies in some areas of 
the province. For example, in the north, almost all 
(99%) of the $13 million spent on temporary special¬ 
ist coverage in 2011/12 was for covering specialist 
vacancies while recruitment was being pursued. 

The Ontario Medical Association also collected 
employment data in a 2011 survey. It described 
some of the barriers new graduates face in finding 
positions in anaesthesiology, cardiac and thoracic 
surgery, general surgery, neurosurgery, orthopaedic 
surgery and vascular surgery: 

• Many cardiac surgeons were working as 
surgical assistants because they could not find 
jobs in their specialty; 34% of those graduates 
who were working as cardiac surgeons still 
considered themselves underemployed. 

• More general surgeons were choosing to do 
fellowships in surgical subspecialties, which 
decreased the number of general surgeons in 
the health-care system. 

• There was competition among orthopaedic 
surgeons for operating room time; older 
surgeons were reluctant to relinquish operat¬ 
ing room time to enable new physicians to 
practise. 

• Senior vascular surgeons were working past 
the normal retirement age, which meant they 
were holding on to operating room time and 
hospital clinic resources. 

Naturally, individual job preferences also affect 
employment patterns. More students maybe 


choosing specialties over general practice because 
of the perception that specialists have more pres¬ 
tige and higher earning potential. According to the 
Future of Medical Education in Canada Postgrad¬ 
uate Project funded by Health Canada, a 50/50 
balance of generalists and specialists is needed to 
provide optimal care to patients. In 2011, Ontario 
had about 1,700 more specialists than general¬ 
ists. The Ministry has worked with the faculties of 
medicine to increase the number of family medi¬ 
cine residency positions by 119% from 2003/04 
to 2011/12. Since 2012, the Ministry and medical 
schools moved to implement a more structured, 
annual planning cycle to better support decision¬ 
making and fine-tuning of the size and composition 
of the postgraduate training system. 

The Ministry acknowledged that graduating 
residents faced a number of employment concerns 
and that unemployment and underemployment 
were concentrated in specific specialties, particu¬ 
larly those requiring hospital resources. However, 
we found that it had not collected data from 
hospitals nor analyzed existing data to identify the 
causes or to develop solutions. For example, the 
Ministry had not examined how hospital funding 
might affect areas such as operating-room capacity, 
and how this in turn might impact employment in 
some surgical specialties. It also had not collected 
data on factors such as the volume of individual 
physicians’ surgical bookings, the allocation of 
surgical bookings among physicians by level of 
experience, or the available operating room cap¬ 
acity across the system. The Ministry indicated that 
it continues to work with stakeholder partners such 
as the Ontario Medical Association and the Ontario 
Hospital Association to better understand how 
profession-specific challenges, including hospital 
operating practices, affect physician employment 
and underemployment. 

The Ministry told us that once the final results of 
the College’s national study are available with juris¬ 
dictional results, it will help inform it of the current 
status of and the multiple factors that contribute to 
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physician unemployment and underemployment 
across Canada. 

Lengthy Wait Times for Specialist Services 

Ministry data that we examined for the three- 
month period from June to August 2013 showed 
long waits for certain surgical services, as shown 
in Figure 4. We found that some of the procedures 
with long wait times were in the same surgical 
specialties in which graduating residents faced 
unemployment and underemployment. For 
example, for some orthopaedic surgeries, waits 
following a specialist’s assessment were as long as 
326 days (forefoot) and 263 days (cervical disc). 
Patients often wait months just to see a specialist 
after the family physician’s referral. 

Our discussions with hospitals we visited and 
the Ontario Medical Association suggested that 
long wait times could be related to factors such 
as hospital funding. Reduced capacity caused by 
budget constraints could mean long wait times 
for some services; if funding is constrained then 
operating room hours and/or the resources who 
staff them could be cut, resulting in unemployment 
and underemployment among the specialists who 
provide these services. 

The hospitals we spoke to in rural areas said 
they needed resources in various specialties, such 


Figure 4: Provincial Wait Times’^ in Surgical 
Specialties with High Unemployment/ 
Underemployment, June-August 2013 

Source of data: Ministry of Heaith and Long-Term Care 


Type of Service/Procedure 

Wait Time (days) 

Neurosurgery (overaii) 

134 

Orthopaedic surgery (overaii) 

192 

Cervical disc 

263 

Forefoot 

326 

Hip repiacement 

186 

Knee replacement 

220 

Lumbar disc 

251 


* Wait time is caicuiated as the number of days from when 9 out of 10 
patients see their speciaiist to when they undergo surgery. 


as neurosurgery, orthopaedic surgery, psychiatry, 
paediatrics, obstetrics/gynaecology, geriatrics and 
emergency medicine, but had difficulty recruiting 
physicians to meet their needs. They indicated 
that graduating physicians often prefer to work 
in large urban centres rather than rural, remote 
and northern areas. Practising in non-urban areas 
presents challenges that may be quite different 
from those encountered during physicians’ medical 
training or posed by practising in an urban centre. 
There are differences in the level of back-up, the 
extent of on-call work and the types of illnesses that 
need treating. There may also be fewer social and 
cultural activities available and limited employment 
opportunities for physicians’ partners. 

Physician Initiatives 

A number of studies have suggested that one factor 
affecting a physician’s practice location decision 
is where he or she receives a significant portion 
of postgraduate medical training. For this reason, 
rural, remote and northern communities may have 
more trouble attracting physicians than urban cen¬ 
tres that are close to medical faculties and teaching 
hospitals. Figure 5 shows a number of programs 
and initiatives the Ministry has funded to help 
those communities recruit and retain physicians. 

The Ministry also manages a Return of Service 
Program that requires international medical gradu¬ 
ates and participants in certain other physician 
postgraduate training programs to practise in 
eligible communities in Ontario, generally for a 
period of five years. (The program is covered in a 
subsequent section of this report.) The Ministry, 
through separate divisions, also funds a number of 
related initiatives, such as the Northern and Rural 
Recruitment and Retention Initiative, which offers 
financial incentives to physicians who establish a 
full-time practice in an eligible northern, rural or 
remote community. 

At the time of our audit, the effectiveness of 
these initiatives had yet to be evaluated. Some of 
the initiatives had only recently been implemented. 
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Figure 5: Selected Ministry Initiatives for Physician Recruitment and Retention in Rural and Remote Communities 


Source of data: Ministry of Health and Long-Term Care 




Funding 
Received in 
2011/12 

1 Initiative 

Description 

($ miiiion) 

Northern Ontario School of Medicine 

Rural-distributed, community-based medical school that seeks to recruit 
students coming from Northern Ontario or rural, remote, aboriginal or 
francophone backgrounds (started 2005) 

12.7 

Distributed Medicai Education 

Organizations co-ordinate clinical teaching placements in small urban 
and rural communities for undergraduate and postgraduate learners 
(started 1995) 

11.7 

Northern and Rurai Recruitment and 
Retention Initiative 

Financial incentives to physicians who establish a full-time practice in an 
eligible community (started 2010) 

3.4 

Hospital Academic and Operating 
Costs for hospitals affiliated with the 
medical education campuses 

Funding to cover academic and operating costs for hospitals affiliated 
with the Medical Education Campuses (started 2008) 

3.5 


The 2010 evaluation of the Return of Service 
Program found that it was not meeting the needs 
of most northern and remote communities. In 
response, the Ministry implemented changes to 
that program, including expanding the eligible geo¬ 
graphic boundaries and providing other targeted 
funding for certain geographic areas that were 
underserviced. 

The Northern Ontario School of Medicine, 
opened in September 2005, has not been operating 
long enough for a meaningful evaluation, but a 
five-year tracking study from 2010/11 to 2014/15 
is under way to determine the extent to which the 
school’s undergraduate and postgraduate programs 
improved the supply and distribution of physicians 
in northern and rural communities. 

In addition, as noted in Figure 5, the Ministry 
funds the Distributed Medical Education Program, 
in which organizations co-ordinate clinical teaching 
placements in small urban and rural communities 
for undergraduates and postgraduates. The Min¬ 
istry informed us that it has been working with 
medical schools since autumn 2012 on an approach 
to evaluate this initiative. 


Physician Locum Programs 

The Agency provides support for temporary phys¬ 
ician or “locum” coverage in eligible communities 
across the province. Locum support targets two 
specific types of need: 

• Respite coverage is an ongoing retention sup¬ 
port to physicians who work in northern and 
rural communities. Because there are fewer 
physicians in these communities, there are 
limited options for local replacements. Respite 
locum coverage provides these rural and 
northern physicians with back-up when they 
are temporarily away on leave, continuing 
medical education or vacation. 

• Vacancy-based coverage is intended as a 
short-term solution to provide access to care 
in areas where there are physician vacancies 
while long-term recruitment is pursued. 

As well as providing access to physician care in 
communities with temporary physician absences or 
vacancies, these programs are also meant to sup¬ 
port the retention of rural and northern physicians. 
In the 2011/12 fiscal year, the Ministry spent a total 
of about $22 million on three physician locum pro¬ 
grams administered by the Agency. 

The programs include the Northern Specialist 
Locum Programs, the Emergency Department 
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Coverage Demonstration Project and the Rural 
Family Medicine Locum Program. 

We focused our audit work on the Northern 
Specialist Locum Programs and the Emergency 
Department Coverage Demonstration Project 
because these programs specifically targeted phys¬ 
ician vacancies while permanent recruitment was 
pursued. 

Costly Long-term Use of Northern Specialist Locum 
Programs 

Northern communities can access up to 26 locum 
specialty services. The Northern Specialist Locum 
Programs incurred $13 million in expenditures in 
2011/12 to provide temporary physician specialty 
coverage through two sub-programs that provide 
short-term coverage for specialist physician vacan¬ 
cies and ongoing respite coverage to support 
retention. 

According to the Agency, the latest available 
data at the time of our audit indicated that about 
30% of specialist positions in Northern Ontario 
were vacant. This translates to a total of about 200 
specialist vacancies, or 40,000 work days that need 
coverage. Data that we examined for the years 
from 2009 to 2011 showed that the specialties 
requiring the greatest number of locum days in 
Northern Ontario were internal medicine, diagnos¬ 
tic imaging, general surgery and psychiatry. Over 
the past five fiscal years, from 2008/09 to 2012/13, 
four large northern cities—Sault Ste. Marie, 
Thunder Bay, Timmins and Sudbury—received 
more than 80% of specialist locum coverage days. 
According to the Ministry, the four larger northern 
city hospitals have the highest usage because they 
act as critical referral centres to the smaller rural 
northern communities where low population and 
other factors would not support specialist practice. 
Also, they provide teaching and research to the 
Northern Ontario School of Medicine. 

In addition to fees for services or claims for 
work sessionals for daily clinical work, payments to 
physicians for vacancy coverage averaged $1,017 
per day for travel, accommodation and honor¬ 


arium—about four times the average amount paid 
for respite coverage. In addition to fees for service, 
physicians receive eligible travel and accommoda¬ 
tion expense reimbursement for respite coverage, 
which amounts to $241 per day on average. In some 
Northern Ontario communities, physician short¬ 
ages and recruitment challenges might have con¬ 
tributed to the extended use of physician locums 
to support ongoing access to care for patients. Our 
review showed that using locums has become a 
service delivery model. Almost all (99%) of the 
$13 million spent on locum coverage in 2011/12 
was for covering specialist vacancies. 

We looked at locum programs in other Canadian 
jurisdictions and found that they generally provide 
only respite coverage. The Ministry indicated that 
Ontario is unique in that it provides large-scale 
hospital-based services to five Northern Urban 
Referral Centres and has a medical school based 
in the north. The mass of critical services in the 
north combined with physician vacancies and 
recruitment challenges in some communities and/ 
or specialties creates a need for locum support that 
may not exist in other jurisdictions. The Agency’s 
long-term goal is to transition to a predominantly 
respite program in Ontario and eliminate use of the 
locum program as a service delivery model. The 
Agency indicated that it is working to implement 
new eligibility criteria, with full implementation by 
2014/15. 

We found extensive reliance on locum programs 
to deliver needed health-care services to some 
rural, remote and northern communities. For 
example, at a number of the hospitals we visited 
we reviewed the on-call locum usage for a single 
month in 2012 and found that locum coverage was 
as high as 94% for internal medicine at one hospital 
and 72% for diagnostic imaging at another hospital. 

Although hospitals with specialist vacancies 
receiving coverage by the Northern Specialist 
Locum Programs are required to post the positions 
on the Agency’s HealthForceOntario jobs website, 
they are not required to report on their progress 
in recruiting for and filling their vacancies. The 
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Agency informed us that it was in the process of 
developing new criteria for locum coverage eligibil¬ 
ity. In a phased approach over the next two fiscal 
years starting in 2013/14, the Agency will require 
hospitals to regularly complete a form, which was 
being piloted at the time of our audit, to inform it 
of recruitment efforts that have been made to fill 
vacancies by specialty area. 

Emergency Department Coverage Demonstration 
Project 

The Ministry and the Ontario Medical Association 
designed the Emergency Department Coverage 
Demonstration Project to be a measure of last resort 
for hospitals to prevent unplanned emergency 
department closures due to physician unavailabil¬ 
ity that otherwise would result in patients being 
unable to access critical emergency services in their 
own community. 

About 50 of 164 emergency departments across 
Ontario have used the project’s locum services since 
it started in 2006. According to the Agency, about 
20 hospitals use this resource at any one time, on 
average. The Agency’s total expenditures on the 
project for the 2011/12 fiscal year were approxi¬ 
mately $4 million. 

We found at the time of our audit that of those 
hospitals using the locum services since before 
January 2008, one-third had been continuously 
using the services from as early as 2007. One 
hospital had been using them from 2006 up to the 
time of our audit. These hospitals received a total of 
about 9,000 hours of locum coverage in 2011/12. 
We reviewed a sample of the monthly emergency 
department schedules between local and locum 
doctors at one of the hospitals we visited and found 
that more than half of the emergency department 
shifts had been covered by locum physicians. 

These findings imply that some hospitals 
facing physician shortages and recruitment chal¬ 
lenges need long-term use of the locum support 
to maintain access to services while permanent 
recruitment is pursued. However, given that 
approximately $4 million was able to cover about 


20 hospitals in meeting the emergency needs of 
communities at any one time, the amount that 
has been spent is not significant overall, averaging 
about $200,000 per hospital. 

Problems With Retaining Physicians in Northern 
and Rural Areas 

The Return of Service Program is intended to pro¬ 
vide greater access to physicians in smaller urban 
communities and underserviced northern and rural 
communities by allowing them to recruit from a 
pool of physicians. The Ministry funds postgraduate 
training and assessment for international medical 
graduates and other physicians seeking to qualify to 
practise in Ontario in return for a commitment from 
them to provide services for a period of time (usu¬ 
ally five years) in an Ontario community (except 
the cities of Ottawa and Toronto and adjacent 
municipalities). As of August 2013, the Ministry 
had active return of service agreements with 550 
physicians who had completed their training. 

According to the Ministry, before 2010 under the 
Underserviced Area Program, physicians could only 
complete their return of service commitment in a 
community designated as underserviced. Origin¬ 
ally, the Underserviced Area Program was focused 
on northern and rural communities. However, over 
time more communities outside of the north were 
designated as underserviced to attract return of 
service physicians, with the result that designated 
communities close to major population centres in 
the south outnumbered those in Northern Ontario, 
which made it even more difficult for northern 
and most rural communities to compete for phys¬ 
ician resources. From 2010 onward the Ministry 
attempted to improve access to care by expanding 
eligible practice areas to include all but the cities of 
Ottawa and Toronto and adjacent municipalities. 
These areas were excluded so they would not be 
able to out-compete smaller urban, rural, northern 
and remote communities for physician resources. 

In our discussions with staff at the rural and 
northern hospitals we visited, we heard repeatedly 
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that they were in need of more physicians. They 
told us that it was challenging to keep return of 
service physicians on after their five-year commit¬ 
ment was up. The Ministry does not keep track of 
international medical graduate physicians who stay 
on and practise in eligible communities after their 
return of service commitments have been met, but 
information we obtained from one of the hospitals 
we visited showed that only one of three physicians 
stayed on after completing their service commit¬ 
ments. The other hospitals we visited did not keep 
information on retention. 

The Ministry has not evaluated the Return of 
Service Program to assess its effectiveness since it 
was redesigned in 2010. There are currently no per¬ 
formance measures or metrics to measure the pro¬ 
gram’s success. The Ministry has only a quarterly 
update on the number of international medical 
graduate physicians in the program, broken down 
by LHIN and specialty at a snapshot date. 

Alternative Health Careers for International 
Medical Graduates 

The Access Centre is a unit within the Agency to 
help physicians and all other regulated health 
professions who want to practise in Ontario with 
the licensing, certification and regulatory process. 
Many international medical graduates use the 
Access Centre for assistance with navigating the 
system and competing for Canadian Resident 
Matching Service (CaRMS) residency positions. 
CaRMS is a national not-for-profit organization 
that provides an electronic matching service for 
postgraduate medical training in Canada. It enables 
applicants to decide where they wish to train in 
Canada and enables programs to indicate which 
applicants they wish to enrol in postgraduate med¬ 
ical training. In the 2011/12 fiscal year, approxi¬ 
mately 2,100 international medical graduates were 
registered with the Agency’s Access Centre, but only 
173 obtained residencies in Canada; of those, 156 
obtained residencies in Ontario. 


Some of the unsuccessful candidates may go on 
to consider a field of practice outside their original 
profession, such as physiotherapy, dietetics and 
chiropractics. In the 2012/13 fiscal year, the Agency 
helped 27 international medical graduates who had 
been unsuccessful in obtaining residencies to tran¬ 
sition to other careers in the health sector. 

Although its current primary mandate does not 
include helping international medical graduates 
find employment in alternative fields, the Access 
Centre informed us that it plans to further develop 
services to assist internationally educated health 
professionals who are unable to practise in their 
field to transition to an alternative health career 
consistent with their education and experience. At 
the time of our audit, the first module of an Alterna¬ 
tive Career Toolkit was available on the Access 
Centre’s website. 


RECOMMENDATION 1 


To better meet the health-care needs of Ontar¬ 
ians, the Ministry of Health and Long-Term Care, 
in conjunction with the HealthForceOntario 
Marketing and Recruitment Agency, should: 

• compare the existing mix and distribution 
of physicians across the province to patient 
needs and consider what measures it can 
take to reduce any service gaps; 

• assess how various factors, including hospi¬ 
tal funding and capacity and health-delivery 
models, affect patients’ access to needed 
services and physician employment, and 
develop cost-effective solutions where con¬ 
cerns are identified; 

• continue to work with medical schools and 
associations to encourage more medical 
students to select fields of study and geo¬ 
graphic areas in which to practise that are in 
demand; and 

• assess the effectiveness of its various phys¬ 
ician initiatives in meeting the health-care 
needs of underserved areas. 


Chapter 3 • VFM Section 3.02 





Chapter 3 * VFM Section 3.02 



2013 Annual Report of the Office of the Auditor General of Ontario 


B MINISTRY RESPONSE 

The Ministry welcomes this recommendation 
as it is consistent with and supports work that it 
has undertaken. Working with the Agency and 
other key health-system partners, the Ministry’s 
HealthForceOntario strategy strives to continu¬ 
ally advance evidence-informed health human 
resource planning that is responsive to the 
health-care needs of the people of Ontario. 

The Ministry will continue working with 
partners to better understand these factors 
and their impact on access to health care and 
health human resource requirements, including 
physicians. 

The Ministry is actively engaged with 
medical schools and professional associations 
to identify and promote a stable supply of phys¬ 
icians across the continuum of specialties with 
a focus on those that are required for harder-to- 
service patient populations or geographic areas. 

The Ministry is committed to ongoing assess¬ 
ment of the effectiveness of initiatives launched 
to address specific needs, and looks forward 
to the outcome of such evaluations to inform 
future planning. 

NURSES 

A comprehensive strategy on nursing was an 
integral part of the HealthForceOntario strategy 
launched by the Ministry in May 2006. The Nursing 
Strategy is a collection of programs and initiatives 
intended to achieve the right number and mix of 
nurses in Ontario now and in the future. It addresses 
issues such as work environments, full-time employ¬ 
ment, and recruitment and retention of nurses. 

At the time the Nursing Strategy was launched, 
the province was experiencing a shortage of nurses, 
and further shortages were anticipated with 
imminent retirements. The Ministry’s focus was on 
increasing the number of nurses—registered nurses 
(RNs), registered practical nurses (RPNs) and nurse 


practitioners (NPs)—in Ontario to improve access 
to care. RNs usually obtain their education through 
a four-year university degree. RPNs usually obtain 
education through a college program. A nurse prac¬ 
titioner is a registered nurse who has acquired the 
knowledge base, decision-making skills and clinical 
competencies for a practice that extends beyond 
that of an RN. 

From 2006 to 2012, the number of nurses 
in Ontario increased by 10%, from 138,583 to 
153,073. The number of nurses employed per 
100,000 people in Ontario increased by 3.5%, from 
1,106 in 2006 to 1,145 in 2012. Figure 6 provides a 
detailed breakdown of the increase. 

We found that the Agency played a small role in 
the nursing sector. Agency regional advisers who 
operate from various LHINs across Ontario focused 
mainly on physicians, and very few Agency advisers 
worked at the Access Centre to counsel internation¬ 
ally trained nurses. 

Although the Nursing Strategy has successfully 
increased the number of nurses in Ontario, we 
found that improvements were generally needed in 
ministry oversight and assessment of the effective¬ 
ness of its nursing programs and initiatives. 

Nursing Initiatives 

According to data from the College of Nurses of 
Ontario, 66.7% of members employed in nursing 
in this province said they worked full-time in 2012. 
The rest were categorized as part-time or casual. 
Figure 7 breaks down the working status for nurses 
in Ontario from 2008 to 2012, and Figure 8 shows 
employment by practice sector for full-time, part- 
time and casual employment, where the nurse does 
not have a set number of hours and is called in to 
work as needed. 

The Ministry has implemented several nursing 
initiatives aimed at stabilizing the workforce and 
increasing full-time opportunities and retention of 
nurses across Ontario. The Health Human Resour¬ 
ces Strategy Division spent $151 million on nurs¬ 
ing initiatives in 2012/13. This amount does not 
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Figure 6: Number of Nurses in Ontario, 2006-2012* 

Source of data: College of Nurses of Ontario 


1 Type of Nurse 

2006 

2012 

Increase 

Increase (%) I 

Registered nurse 

108,185 

112,194 

4,009 

4 

Registered practicai nurse 

29,706 

38,859 

9,153 

31 

Nurse practitioner 

692 

2,020 

1,328 

192 

Total 

138,583 

153,073 

14,490 

10 


* The data provides a “point-in-time" snapshot of the avaiiable iabour suppiy of nurses {at the end of the 
prior year or beginning of the stated year). 


Figure 7: Overaii Working Status of Nurses in Ontario, 2008-2012* 

Source of data: College of Nurses of Ontario 



2008 

2009 

2010 

2011 

2012 

1 Working Status 

# 

% 

# 

% 

# 

% 

# 

% 

# 

% 

Fuii-time 

75,649 

62.9 

78,694 

63.9 

80,356 

63.8 

83,972 

66.4 

85,010 

66.7 

Part-time 

34,820 

29.0 

34,371 

27.9 

34,939 

27.8 

32,316 

25.6 

32,712 

25.6 

Casuai 

9,796 

8.1 

10,026 

8.2 

10,549 

8.4 

10,117 

8.0 

9,889 

7.7 

Total 

120,265 

100.0 

123,091 

100.0 

125,844 

100.0 

126,405 

100.0 

127,611 

100.0 


* The data provides a “point-in-time” snapshot of the working status of nurses (at the end of the prior year or beginning of the stated year). 


Figure 8: Empioyed Nurses^ by Practice Sector, 2010^ 

Source of data: College of Nurses of Ontario 




1 

Registered 




1 Registered Nurses 

Practical Nurses 

Total 


Working Status 

■ • 

% 

# 

% 

# 

% 

Full-time 

Hospitai 

41,030 

44 

7,544 

25 

48,574 

40 

Long-term care 

4,851 

5 

6,690 

22 

11,541 

9 

Community 

10,947 

12 

2,636 

9 

13,583 

11 

Other & not specified 

4,656 

5 

771 

2 

5,427 

4 

Part-time 

Hospitai 

15,576 

16 

4,669 

15 

20,245 

16 

Long-term care 

2,201 

2 

3,597 

12 

5,798 

5 

Community 

4,757 

5 

1,409 

4 

6,166 

5 

Other & not specified 

2,052 

2 

468 

1 

2,520 

2 

Casual 

Hospitai 

4,352 

5 

1,168 

4 

5,520 

4 

Long-term care 

552 

1 

804 

3 

1,356 

1 

Community 

2,134 

2 

506 

2 

2,640 

2 

Other & not specified 

808 

1 

180 

1 

988 

1 

Total 

93,916 

100 

30,442 

100 

124,358 

100 


1. Not including nurse practitioners. 

2. Most recent year that data was available. 


Chapter 3 • VFM Section 3.02 




































Chapter 3 * VFM Section 3.02 


2013 Annual Report of the Office of the Auditor General of Ontario 


include funds spent by other program areas within 
the Ministry for nursing initiatives. 

We examine a few of the Ministry’s more signifi¬ 
cant initiatives in the following sections. 

Meeting the 9,000 Nurses Commitment 

In 2007, the government committed to hiring 9,000 
more nurses over a four-year period. It also com¬ 
mitted to a goal of 70% of nurses working full-time. 
However, in fall 2008, the province indicated that it 
would take longer to achieve the goal of 9,000 new 
hires. From 2008/09 to 2011/12, $309 million was 
dedicated to this initiative. Our review of the initia¬ 
tive showed that: 

• From year-end 2007 (reported as 2008 by the 
College of Nurses of Ontario), the number 

of nurses in the province increased by more 
than 7,300 over four years. At the time of our 
audit, CNO nursing data was not available for 
2013 (which would represent 2012 figures), 
but it appeared likely that the goal of hiring 
9,000 new nurses would be achieved by the 
end of 2012. 

• The province was slightly under its goal of 
having 70% of nurses working on a full-time 
basis. As of the year ended 2011, reported by 
CNO as 2012 numbers, 66.7% of nurses were 
working full-time. From the year ended 2007 
(reported as 2008) to the year ended 2011 
(reported as 2012), there were almost 9,400 
more nurses working full-time, representing a 
12% increase. 

We reviewed five Nursing Secretariat programs 
that reported creating 1,316 nursing positions. Two 
of the programs we reviewed had been in place 
since 2008/09 and the other three had been imple¬ 
mented in 2011/12. The Ministry indicated that 
1,125 of the 1,316 new positions (85.4%) had been 
filled by March 2013. We also noted that the 1,316 
nursing positions created were not all full-time, but 
included part-time and casual positions. The Nurs¬ 
ing Secretariat did not have detailed information 
to determine the type of employment obtained by 
some of the positions funded through the Division. 


Specific data was not available to determine the 
number of full-time, part-time or casual positions 
that the funded organizations had created through 
the 9,000 Nurses Commitment. 

Inadequate Assessment of the Nursing Graduate 
Guarantee Program’s Effectiveness 

Announced in February 2007, the Nursing Graduate 
Guarantee Program’s objective was to support new 
Ontario nursing graduates (both RNs and RPNs) 
in finding full-time employment immediately upon 
graduation. Some of the program’s other objectives 
include facilitating recruitment in all nursing sec¬ 
tors; transforming employer practices to make more 
full-time nursing positions available; and increasing 
the total supply of nurses by providing full-time 
employment to nurses who may have otherwise 
sought work in other jurisdictions or professions. 

The program provides funding for temporary 
full-time, above-staffing-complement positions for 
26 weeks with the expectation that these bridg¬ 
ing positions will lead to permanent full-time 
employment. Employers must commit to funding 
an additional six-week full-time position for the 
new graduate nurse if he or she is not bridged to 
permanent full-time employment by the end of 
the 26 weeks. The program is open to employers 
in all health sectors (hospitals, long-term-care 
homes and community care organizations); all 
Ontario-educated new graduate nurses are eligible 
to take part as long as they register on the Agency- 
administered new graduate guarantee job website 
and accept a job offer within six months of complet¬ 
ing their studies. In 2011, there were approximately 
1,200 potential employers. In the 2011/12 fiscal 
year, about $66 million of ministry funding was 
provided to about 210 participating health-care 
organizations (representing an 18% employer par¬ 
ticipation rate), which employed 2,235 new gradu¬ 
ate nurses under the program. 

According to the Ministry’s January 2011 
Guidelines for Participation in the Nursing Gradu¬ 
ate Guarantee for New Graduate Nurses, program 
funding is provided with the expectation that the 
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bridging positions offered by organizations will 
lead to permanent full-time jobs. In the transfer- 
payment agreements that they enter into with the 
Ministry, participating organizations commit to 
making their “best effort” to place a new graduate 
in a permanent full-time position after he or she has 
worked with the employer for at least 12 weeks, but 
the Ministry does not assess program data against 
established program targets to determine how well 
organizations are doing with bridging participants 
to permanent full-time employment. We conducted 
our own analysis of 2010/11 and 2011/12 program 
data and found that only about one-quarter of 
participants had been transitioned to permanent 
full-time employment after the six-month period 
in 2010/11 and one-third in 2011/12; about an 
additional one-third of participants had been tran¬ 
sitioned to part-time employment in both 2010/11 
and 2011/12. 

We also found that the Ministry had not 
reviewed employment trends at participating 
health-care organizations to determine whether 
employers were making their best efforts to transi¬ 
tion program participants to full-time permanent 
employment. We reviewed a sample of health-care 
organizations that received funding from the 
program in 2007/08 to 2011/12 to look for employ¬ 
ment patterns and found the following: 

• In 2011/12, one organization reported that 
15% of its graduates had transitioned to 
full-time employment while 65% went to part- 
time. In 2007/08, the same employer reported 


24% of its graduates had transitioned to full¬ 
time employment and 59% to part-time. 

• In 2007/08, another organization reported 
that 85% of its graduates had transitioned 
to full-time employment and 2% went to 
part-time. By 2011/12, the same organization 
reported that 19% of its graduates had tran¬ 
sitioned to full-time employment and 47% to 
part-time. 

• In 2011/12, an organization reported that 
40% of its nursing graduates had voluntarily 
left the program early. Sixty percent of them 
left because they found employment else¬ 
where, while no explanations were given for 
the others who left. 

The Ministry has allocated funds ranging from 
about $86 million to almost $100 million per year 
for the program since it began in 2007/08. We 
found that for each of the past five fiscal years 
(from 2007/08 to 2011/12), program expenditures 
were less than the amount of funds allocated. As 
shown in Figure 9, other than for 2009/10, the 
amounts of unspent funds varied considerably, 
ranging from $17.2 million (20%) to $33.6 million 
(37%) of total funds allocated for the year; they 
totalled $105.7 million over the five-year period. 
Figure 9 also shows how declining program 
expenditures are related to a one-third drop in 
program participation by eligible graduate nurses. 
Participation was as high as 62% in 2007/08 but 
declined to 35% in 2011/12. When we asked 
the Ministry about the reasons for this decline. 


Figure 9: Nursing Graduate Guarantee Program Funding and Participation 

Source of data: Ministry of Heaitii and Long-Term Care 



Allocated 

Actual 

Unused 

1 

# of New Nurse 

% of New Nurse 

Amount 

Expenditure 

Portion 

#of New Nurse 

Graduates 

Graduates 

Fiscal Year 

($ million) 

($ million) 

($ million) 

Graduates 

Participating 

Participating 

2007/08 

88.9 

71.7 

17.2 

4,300 

2,660 

62 

2008/09 

94.2 

72.3 

21.9 

4,902 

2,825 

58 

2009/10 

85.8 

85.5 

0.3 

5,139 

2,598 

51 

2010/11 

87.5 

54.8 

32.7 

5,555 

1,804 

32 

2011/12 

99.6 

66.0 

33.6 

6,386 

2,237 

35 

Total 

456.0 

350.3 

105.7 
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it indicated that not every nursing graduate will 
secure a job through this program. Some move 
into positions in the same organization where they 
completed their clinical placement, but outside the 
program, while others may decide to continue their 
studies or take time off. Nevertheless, the signifi¬ 
cant decline in graduate participation in this pro¬ 
gram raises questions that need to be considered in 
evaluating the program’s effectiveness. 

An external party has evaluated the Nursing 
Graduate Guarantee Program annually since 
2007/08 using ministry data supplemented by a 
survey of participants and employers. The 2011/12 
evaluation resulted in nine recommendations that 
included, for example, continuing to promote the 
participation of long-term-care and community 
employers in the program; supporting the partici¬ 
pation of northern, rural and small organizations 
in the program; and examining the differences in 
employment status, retention and transition into 
the nursing profession across sectors. The Min¬ 
istry indicated that it was working to address the 
recommendations. 

However, the evaluator has not assessed the 
overall effectiveness of the program. For example, 
it has not reported on the total percentage of nurse 
graduates who have transitioned to permanent full¬ 
time employment through the program. 

The Ministry has set performance targets for 
the number of temporary full-time positions to be 
bridged—the number of participants—but has not 
set an outcome-based performance target— the 
number of participants to achieve permanent full¬ 
time positions—for the Nursing Graduate Guaran¬ 
tee Program. In the program’s first year (2007/08), 
the target was 3,000 positions to be bridged; by 
2011/12 the target had decreased to 2,500 pos¬ 
itions. The Ministry met its targets in only one of 
the five years—2008/09. 

Inadequate Assessment of Nurse Practitioner- 
led Clinics 

The purpose of nurse practitioner-led clinics is to 
provide increased access to primary health care to 


the people of Ontario. Patients at these clinics see 
a nurse practitioner (NP) as their primary health 
care provider, consulting with a physician only 
when needed. The first clinic opened in Sudbury in 
2007 and served as a pilot project for the initiative. 
In November 2007, the Ministry announced that 
it would establish 25 more NP-led clinics across 
the province. In 2011/12, the 26 clinics received 
$29 million in ministry funding for development 
and implementation. 

Each clinic can have up to four NPs who operate 
in collaboration with an interprofessional team 
(such as RNs, dietitians and social workers) to 
provide increased access to primary health care. To 
achieve this, the clinic may, for example, focus on 
providing: 

• family health care for people previously with¬ 
out a primary care provider; 

• chronic disease detection and management, 
such as obesity programs, smoking cessation 
and cancer screening; and 

• faster access to care through house calls and 
same-day or next-day appointments and 
extended hours. 

We looked at information from a sample of clin¬ 
ics to determine whether the majority of patients 
had indeed previously been without primary care 
providers. We found that only two of the five clinics 
we contacted had taken steps to document whether 
their patients had family physicians. Two of the 
other clinics informed us that patients were asked 
if they had a family physician. The remaining clinic 
did not begin collecting this information until April 
2013, when it began having patients complete an 
intake form that included a question about family 
physicians. 

At the time of our audit, the Ministry’s overall 
target was to have a total of 40,000 registered 
patients (who do not have regular access to a family 
health-care provider) at all 25 clinics. As of January 
2013, the 24 NP-led clinics that were open (one NP- 
led clinic is targeted to open late in the 2013/14 fis¬ 
cal year) reported having about 33,000 registered 
patients, or 82% of the program target. Given that 
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the clinics are a new model of primary health-care 
delivery and that many of them had been open less 
than two years, not all clinics are at full capacity, as 
it takes time to establish a patient roster. 

Clinic budgets setting out operational and one¬ 
time costs are approved annually by the Ministry. 
Clinics are required to submit quarterly and annual 
financial and performance information and statis¬ 
tics to the Ministry for review as well as an audited 
statement of revenues and expenditures for the 
year. Our review of ministry documentation from a 
sample of clinics indicated that the Ministry gener¬ 
ally followed up with clinics on matters during 
their development phase. Ministry staff informed 
us that they reviewed clinic operating costs, com¬ 
paring actual operating expenditures to those set 
out in the clinic’s budget, approving funding levels 
and discussing any variances with the appropriate 
clinic staff. 

The Ministry’s performance measure for 
this program is the establishment of 25 nurse 
practitioner-led clinics, which has been met. 
However, this measure does not assess whether the 
clinics are effectively meeting program goals. At 
the time of our audit, the Ministry had evaluated 
only one clinic (Sudbury) for its effectiveness in 
meeting program objectives. 


RECOMMENDATION 2 


To provide an appropriate level of nursing ser¬ 
vices and thereby improve access to care across 
the health sector, the Ministry of Health and 
Long-Term Care should: 

• monitor nursing employment trends and 
assess the outcome of its nursing initiatives 
in transitioning graduating nurses to perma¬ 
nent full-time employment; 

• assess the reasons for declining participa¬ 
tion rates of nurse graduates in its Nursing 
Graduate Guarantee Program, and take steps 
to improve program effectiveness, including 
encouraging participation in the program 
across sectors; and 


• monitor the nurse practitioner-led clinics 
more closely to ensure that they are meeting 
program requirements and achieving their 
patient targets and program objectives. 

B MINISTRY RESPONSE 

The Ministry acknowledges the recommenda¬ 
tion and comments regarding the Nursing 
Graduate Guarantee (NGG) program and the 
nurse practitioner-led clinics. 

Recognizing the importance of monitoring 
and evaluating the NGG program, work has 
been under way since early 2012 to enhance its 
online management. A new tool was launched 
in April 2013 that allows the Ministry to collect 
and analyze information that was previously 
unavailable, including monitoring aspects of 
NGG participation rates by both nursing gradu¬ 
ates and employers, as well as employment 
outcomes. 

The annual program evaluation will con¬ 
tinue and will be enhanced to examine the 
variations in participation rates. 

The Ministry will also implement a targeted 
communication strategy to promote increased 
uptake of NGG participants across health-care 
sectors, with particular attention to the com¬ 
munity—for example, in home care, long-term 
care, primary care and public health sectors. 

The recommendation about nurse 
practitioner-led clinics is timely as the clinics 
transition from startup to full operations. 

The Ministry will review how it can apply 
greater oversight to these clinics and ensure 
accountability for outcomes including 
achievement of patient targets and program 
objectives. The Ministry will continue to 
take timely, appropriate action when non- 
compliance with agreements is identified. 
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Untimely Recovery of Unspent Funds 

Organizations that receive transfer payment fund¬ 
ing from the Ministry are required to submit annual 
financial statements. Subsequent to year-end, the 
Ministry reviews year-end financial statements to 
assess whether it is owed any surplus funds. Any 
payables to the Ministry are recovered from the 
transfer payment recipients. 

For the nursing initiatives we examined in our 
audit, we found that some related transfer payment 
agreements did not set out time requirements for 
submitting or completing the financial reconcilia¬ 
tion. For some organizations that received fund¬ 
ing through the 9,000 Nurses Commitment, the 
information provided was lumped into broader 
programs instead of being broken down by initia¬ 
tive. For example: 

• Our review of a sample of the organizations 
that received 9,000 Nurses Commitment 
funding found 36 programs for which many 
of the year-end reconciliations had not been 
completed on a timely basis. 

• At the completion of our audit, the Ministry 
had completed reconciliations up to only 
2009/10 for the Nursing Graduate Guarantee 
Program. Pending completion of the other 
years, the total amount of recoveries identi¬ 
fied to date was at least $7.3 million. The 
Ministry was still in the process of recovering 
the funds. 

• We looked at a sample of five nurse 
practitioner-led clinics and identified total 
operating funds of about $1.3 million owed to 
the Ministry for the 2011/12 fiscal year. This 
amount represented about 30% of the total 
funding provided to these clinics. In addition, 
the audited statements of two of the clinics 
showed about $360,000 owing to the Ministry 
for the 2010/11 fiscal year. $hortly after we 
completed our audit fieldwork, the Ministry 
indicated that it had identified approximately 
$3.4 million in recoveries related to 2009/10 
and 2010/11, which it was in the process of 


recovering from the clinics. The Ministry also 
informed us that it is working to complete the 
reconciliations that remained for 2009/10, 
2010/11 and 2011/12. 

In October 2012, the Ministry of Finance’s 
internal audit department issued a report on 
oversight and monitoring of transfer payment 
recipients that made observations similar to ours. 
It recommended establishing outcome-based 
performance measures and guidelines for review 
and analysis of financial reporting to increase 
consistency and enhance efficiency. The Ministry 
informed us that the focus of the strategy to date 
has been to establish capacity in the province by 
increasing the supply of providers, which relates 
to an output-based measure. It has plans to estab¬ 
lish outcome-based performance measures and 
guidelines where appropriate. 


RECOMMENDATIONS 


To improve financial oversight of funded 
organizations and recover unspent funds, the 
Ministry of Health and Long-Term Gare should 
perform timely reviews of relevant financial 
statements. 

B MINISTRY RESPONSE 

The Ministry is committed to enhancing finan¬ 
cial management systems to ensure optimal 
use of financial resources. In 2013/14, the 
Ministry is implementing processes that will 
ensure emphasis is placed on in-year review 
and analysis of financial reports submitted by 
funding recipients to support timely recovery or 
payment adjustments. 

For the HealthForceOntario strategy, the 
Ministry has made significant progress over the 
past two fiscal years through dedicated efforts in 
the area of reconciliations. Its goal is to have all 
reconciliations current by March 31, 2014. 
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HEALTH HUMAN RESOURCE 
FORECASTING MODELS 

Forecasting models are recognized as one import¬ 
ant component of evidence-based health human 
resource planning. Good information and proper 
health human resources planning are essential if 
the Ministry and health system stakeholders are to 
work together to determine an appropriate number 
and mix of health professionals to meet the health 
needs of Ontarians across the province. 

Better Physician Forecasting Data Needed 

Some Canadian jurisdictions are engaged in 
physician forecasting and modelling. For example, 
Alberta and Nova Scotia have developed needs- 
based physician forecasting models similar to 
Ontario’s to plan their physician supply require¬ 
ments. This type of planning generally involves 
estimating the health services required to meet the 
needs of the population and then translating them 
into the number and type of physicians required to 
deliver those services. 

Historically, physician human resources plan¬ 
ning in Ontario has been supply- or utilization- 
based; however, this method does not provide a 
complete picture because it does not consider the 
population’s health needs. In 2009, the Ministry 
partnered with the Ontario Medical Association 
and used a tendering process to select an external 
party to develop a new, needs-based model. The 
new model works by examining the population’s 
health needs and translating them into needs for 
physician services, then comparing these needs to 
the supply of physician services currently available. 
Service gaps are quantified and converted into the 
number of physicians required to meet the needs. 

Ontario’s physician forecasting model is a 
positive step in determining physician workforce 
requirements. However, the model is hampered by 
the limited reliability and availability of data. We 
found that Ontario’s model does not account for 
some important variables because of a lack of avail¬ 


able quantifiable data on physician productivity. 
Physician productivity is an important component, 
and even small improvements in productivity can 
have a significant impact on the number of human 
resources required in the system. 

During development, the consultant who built 
the model defined productivity as the number of 
patients seen in the physician’s practice for a given 
period of time, noting several factors that affect 
physician productivity, including: 

• information and communication technology 
(electronic health records, telemedicine); 

• health system change (new or different 
primary health-care models, such as Family 
Health Teams); 

• non-physician clinicians (other health-care 
providers working with physicians, such as 
NPs); and 

• funding and compensation models. 

Even for these four factors, specific quantifiable 
data for only one category of non-physician clin¬ 
icians—NPs—was available and incorporated into 
the model. 

Also, while physician human resource forecast¬ 
ing reflects factors such as workforce demographics 
and changing population health needs, it is also sig¬ 
nificantly affected by broader economic, social and 
health-system trends, as well as health technology 
advancements. Many of these other factors—the 
availability of diagnostic and laboratory equipment, 
operating-room time and space to perform surger¬ 
ies, physician preferences for certain specialties and 
practice locations, and employment opportunities 
across the province—can affect access to and 
delivery of health-care services, but they are not 
easily incorporated into the model. Although the 
needs-based simulation model does make it pos¬ 
sible to test “what-if ’ simulations that may help to 
assess the impact of some of these factors, it cannot 
incorporate all of them. 

For these reasons, the results obtained from 
Ontario’s physician forecasting model can be 
considered only one of many tools and pieces of 
evidence available to support policy formulation. 
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The model’s limitations make it difficult to use in 
planning for the optimal number, mix and distri¬ 
bution of physicians with appropriate funding, 
training and deployment across the province. 

Lack of Forecasting of Demand for Nursing 
Services 

In 2008, the Ministry also engaged an external 
consultant to develop a needs-based nursing 
model that would be able to project the gap 
between the need for and the actual supply of 
RNs, RPNs and NPs in Ontario for each year over a 
10-year period. Separate simulation models to test 
various health human resources policy scenarios 
were also developed for both RNs and RPNs. 

Although the model cost about $435,000 to 
develop, it was initially not used to specifically 
inform any nursing policies because of concerns 
about the accuracy of its predictions. 

In 2012, as part of its ongoing evidence 
development work, the Ministry found that the 
model had understated data for first-year enrol¬ 
ment of RNs in 2007 and incorrectly assumed 
that all RNs provided the same rate of direct 
patient care regardless of their years of experi¬ 
ence; in practice, younger RNs just entering the 
profession may be providing different amounts 
of direct patient care than RNs near the end of 
their careers. The model had also applied an 
estimated percentage of total RNs providing direct 
patient care based on previous outdated data. 

The Ministry corrected the first-year enrolment 
data and updated the direct patient care data and 
completed two simulations. However, assump¬ 
tions regarding attrition, retirement and workload 
were not updated, and no other data reviews were 
conducted. 

The model has other limitations. For example, 
it forecasts the gap between supply and need at 
the provincial level but not at a regional level. 

At the time of model development, data on the 
patterns of service delivery for NPs was not avail¬ 
able, and therefore the consultant was unable to 


conduct a simulation model analysis that would 
accurately estimate the requirements for this nurs¬ 
ing group. 

During the course of our audit, the Ministry 
was in the process of working with the Ministry 
of Training, Colleges and Universities to develop 
a supply-based nursing simulation model for RNs 
and RPNs, with results expected to be available by 
late 2013. 

The Ministry advised us that the purpose of 
the supply-based simulation model is to help 
the government plan properly for future nursing 
education positions and formulate nursing recruit¬ 
ment and retention policies. The initial model 
is to provide projections for the future supply of 
nurses (RNs and RPNs) at the regional level and 
will enable planners to test the potential impact of 
policy changes on the supply, such as changes to 
enrolment numbers and percentages of full-time 
nurses, the introduction of incentives for working 
in rural and remote areas, shifting distribution 
of nurses by employment sector, recruitment 
and retention rates and other attrition factors. 
However, a supply-based model cannot assess 
whether the supply is appropriate because it does 
not take into account the population’s need for 
nurses. In addition, the new supply-based model 
will not include NPs because some historical data 
specific to them is not available from the College 
of Nurses of Ontario. The Ministry informed us 
that the initial focus is on the future supply of 
nurses, which is important work that needs to be 
completed to support future needs-based model¬ 
ling considerations. 

To get better information to inform future 
policy work, the Ministry entered into an agree¬ 
ment with a large hospital in late 2012 to create a 
one-time snapshot of the current supply, distribu¬ 
tion and predicted shortfall or surplus of RNs, 
RPNs and NPs working in selected primary health¬ 
care organizations, long-term-care homes and 
acute-care hospitals across the province for the 
next three months. Each surveyed health organ¬ 
ization is to report on current staffing, vacancies. 
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details about leaves of absence and predictions of 
short-term staffing changes. There will also be a 
comprehensive analysis of the overall staffing situa¬ 
tion in each area. This data will be used to identify 
which organizations and geographical areas are 
having difficulty in recruiting and retaining nurses 
in Ontario. 


RECOMMENDATION 4 


To provide reasonable and reliable forecasts 
of the requirements for physicians and nurses 
and to better ensure effective health human 
resources planning, the Ministry of Health and 
Long-Term Care should: 

• conduct assessments of employment trends, 
the supply and projected needs for health 
services, and the associated health workforce 
requirements to best meet those needs cost- 
effectively; and 

• for physicians and nurses, further refine its 
forecasting models and their capabilities 
to assess the impact of various factors on 
service-provider productivity. 


B MINISTRY RESPONSE 

The Ministry agrees with the audit findings that 
its physician forecasting model is a positive step 
in determining physician workforce require¬ 
ments. Similarly, the Ministry concurs with the 
audit observations regarding forecasting models 
as one of many types of tools that are required 
to support health human resource planning. 

As such, the Ministry is actively engaged 
with the health sector to improve evidence for 
decision-making. Over the summer of 2013 
the Ministry has been meeting with the field 
regarding the current health human resource 
environment, including how we continue to 
evolve and develop evidence. This work will 
contribute important information to inform 
future HealthForceOntario work. 

The HealthForceOntario strategy will 
continue to provide innovative health human 
resource solutions to meet patient-care needs. 
Evidence for decision-making will continue to 
be a key aspect of the strategy and the Ministry 
will seek to enhance and expand tools, including 
forecasting models, to improve planning. 
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